Revelations of the mismanagement at the now-bankrupt Japanese bitcoin exchange keep surfacing. When laying the puzzle as pieces keep coming, it becomes obvious that security at the billion-dollar vault was practically nonexistent. This adds to previous insights of economic and/or fraudulent mismanagement.
An interesting blog post from Mark Karpeles resurfaced recently. Mr. Karpeles was the CEO of the now-imploded Japanese bitcoin exchange MtGox, nicknamed “Empty Gox” for its previously-rumored insolvency. The blog post reveals a stunning ignorance of the concept of security, going beyond nonexistent security and into daredevil-reckless territory.
Jacob Appelbaum, the world-class security researcher and one of the spokespeople for the anonymity service Tor that has saved many activist lives worldwide, tweeted sarcastically about the article:
I think this is perhaps the most amusing technical blog post I've read in ages: http://t.co/uJwtz5xtrU
— Jacob Appelbaum (@ioerror) February 28, 2014
The article in question (gone from the server, but saved by the Internet Archive) was about how Karpeles had decided to write his own security mechanisms for remote access to his core servers. This goes against every grain, every practice, every professionalism of good security that exists. Security is hard and needs thousands of eyes to find the small but important bugs – just last week, a bug in Apple’s iOS was discovered where an attacker could have impersonated any target. And that was from Apple.
Any person who calls themselves a professional in the IT field will end the conversation with anybody, no matter what title, who boasts that they have created their own security. You just don’t do it. It’s beyond reckless. It’s practically a guarantee that you will get broken into tracelessly.
It gets worse. Karpeles didn’t just write his own remote-access security (“SSH server”). He did so in the programming language PHP, which is a dangerously unsafe language intended for low-security applications like displaying web pages. It basically has no error checking or safety nets of any kind. So not only did Karpeles think it was a good idea to do something that almost guaranteed MtGox to get hacked, he did so using one of the worst possible tools imaginable. It wasn’t enough to shoot himself in the foot and reload, he had to pick a bazooka to do it.
(UPDATE: As some have pointed out, this is no definite proof said home-cooked SSH server written in PHP was used as production code on Gox. This observation is correct. However, the primary observation here is the reckless disregard for security. This is further accented by three more observations: first, in the comments in the article, Karpeles states that he intends to wrap this PHP SSH code into a production library, and second – quote in same comment field – “RSA re-implemented in pure PHP is not a bad thing.” The third observation is that a commenter named Nanashi pointed out the PHP SSH server as “the least secure implementation ever” in 2010, and while we still don’t know if it was run at Gox in production for remote server access, it’s a rather striking coincidence that the same name – Nanashi – was behind an the enormous database leak from Gox’ internal databases described later in this article.)
This is not professional behavior. This is completely-over-the-top amateurish, from somebody who a) doesn’t understand security at all and b) is so convinced of their own perfection that they dismiss every criticism. People are even pointing out flaws in his implementation in his own comment field, and he just dismisses it, despite the fact that these flaws would be enough for an adversary to assume remote control of his core servers – “ownage”, as it is called.
Traditional SSH servers too secure for you? Write one in PHP! http://t.co/v2in7x6NJd
— Kyle Steely (@modalexii) February 28, 2014
When you read these facts, if you understand security, your hairs stand on your arms, you are pushing away from the screen in balking disbelief, and your eyes are going wide. This guy had taken on safekeeping of a billion dollars for his clients?
Let’s be clear: To anybody who is the slightest aware of good security practices, this article from the principal architect of the bitcoin exchange is not some government-issue red flare going off in the corner of your eye. This is a goddamn Betelgeuse going off.
To put it in non-technical terms, this is roughly somebody who claims they are qualified to be a heart surgeon because they have read the back cover of “Anatomy for Dummies”. Not just that, but they actually open a heart surgery practice.
It’s like asking for a hardened veteran infantry officer to lead a batallion into battle, and having a random guy who has read military comic books show up for the task.
It’s like asking if somebody can build a complex skyscraper, and somebody shows up with a grin from ear to ear explaining that they already found everything they need for the job in trash bins on the way to the meeting.
Somebody who was utterly not qualified to go near any kind of security job had built a vault for a billion dollars using a completely unsafe webpage scripting language. And people were using it, trusting him with their money, more or less because he said he was honest in the Terms of Service.
It gets worse. The forensic site MtGoxProtest has an interesting inside view of security practices at the company that stored bitcoin, US Dollars, and euros for its clients to a value of about a billion dollars (at peak bitcoin value).
If the product was so thorougly shoddy in terms of security, some very skilled staff at some very skilled companies are able to mitigate that by rigorous processes and consistent pride in their work. What about Gox? How did they relate to security in their daily work?
They didn’t give a shit.
Security alarms would go off, somebody would notice something totally alarming, and they would basically just ignore it.
On the surface, security looked decent. Clients would log in using two-factor authentication, not relying on a hackable password alone. Clients could separate withdrawal security from authentication security, adding a second security layer when they wanted to get money out of their account.
(This is disregarding all jokes that you couldn’t actually get any money out of the vault, because it was empty – hence “Empty Gox”.)
But it happened much too frequently that client accounts were emptied anyway overnight, and provably so by somebody else than the account holder. This should have set of major alarm bells at the Gox offices; somebody was apparently and obviously able to circumvent their security layers and access the servers directly. Mere suspicion of that is cause for a total shutdown until forensics have cleared out what the hell happened.
So what happened?
They didn’t give a shit. They blamed the customers and went about their daily business.
Coupled with the above article from Karpeles – who wrote much of the initial Gox codebase – about how Gox would violate every security practice in existence and then invent some more just so they too could be violated, it becomes clear that the strict login procedures were just for show. Gox was leaking like a bloody sieve, and Karpeles was too incompetent and too proud to understand the magnitude of the disaster in the making.
According to the insiders’ information, security researchers would regularly submit alarming reports of gaping security holes that would just as routinely be completely ignored. And then security researchers did what they do when companies ignore them, which is publish their findings. So now there was not only a billion-dollar vault with security holes the size of the Empire State Building, there were also published research papers on where they were and how they worked.
And Gox? They continued to not give a shit.
It gets worse. They treated business processes the exact same way as they did security processes: “It seems to run well and we don’t really care”. I have hinted in my previous posts that I’ve got stuff that would be jawdropping; I’m not sure it is anymore, it’s just in line with the total mismanagement – no, fraudulent operations – that has been going on. They basically didn’t know anything about contract or finance business, either.
My specific case was that I had been offered X bitcoin for Y US Dollars on the exchange webpage, clicked “buy”, and even got a separate confirmation box: “Do you want to buy X bitcoin for Y US dollars?”. As I clicked “Yes” to that, that’s entering into a legally binding contract. But I wasn’t delivered X bitcoin – I was delivered X-56 bitcoin for the Y USD, which is a rather large difference. As I pointed this out to support, that they had an unfulfilled obligation of 56 bitcoin to me, they explained patiently how the quote price was calculated from a technical standpoint, why I had been charged a much higher price than quoted, and implied that the technology and interface were working just as designed.
Listen here Karpeles, I don’t care in the slightest why you think the price should be higher than quoted – if you quote a price and I accept the offer, you deliver on it, and you deliver exactly what was offered. If you can’t do so, that’s your problem, not mine.
They didn’t understand these very basics of running a business. They didn’t understand the concept of offers, accepts, and contracts. Or they just didn’t care. This 56-coin claim of mine was one of the open items in support threads when Gox folded, and I was totally prepared to bring that to court. Now it’s rolled up in my overall bankruptcy claim instead.
It gets worse.
Yesterday, a leak was posted with internal accounting data at MtGox. It contained every customer balance, their last login timestamp, withdrawal limits for every customer, and a lot of other client data. Whoever orchestrated that leak had access to the internalmost servers at Empty Gox.
But just to drive the point home, the damning leak was posted from Mark Karpeles’ personal accounts, both on Reddit and in an article on his own personal blog.
That is such total ownage of somebody’s poor security, there’s nothing left to say. It’s confirmation of everything from the original article – that this is a person who didn’t understand the most basic security practices.
Gox appears to have been run on the kind of security that only an idiot would have on their luggage.
Good riddance!
(I mean no disrespect for those of us that have lost funds at MtGox)
OK Rick we get it that you are utterly pissed about losing a lot of virtual profit. But hey who said it should be easy living the life of an ultra capitalist! As you proclaim to be.
If only you could read and actually see that the blog post from mid 2010 has nothing to do with MTGox…
1) It’s not any better to have that kind of non-security at a hosting company.
2) Coders reuse code. That’s how coders build a codebase. If Karpeles solved this problem in this way in his previous company, and it seemed to work well there, then that’s how he will solve the same problem again. Odds are he’ll just bring along the old code, even: that’s how it’s typically done if the code is yours.
> my overall bankruptcy claim […]
Are you claiming bankruptcy?
No – I was referring to my claim against the bankruptcy proceedings of Gox. As disclosed before, I lost holdings against the empty vault.
Good summary Rick. I’m speechless
Rick, I need some help understanding. Either Mt Gox was hacked and all the bitcoins were stolen because of glaring incompetence by the staff, or it was an inside job and the hacking was just a coverstory. Your posts seem to suggest it is both. Please explain what I’m to believe now.
Thanks
Hailaga
It is starting to add up, isn’t it?
We know that practically everything Karpeles has said has been a lie, intended to stall, deflect, or outright disinform. Being unprofessional is one thing. But these are the signs of a scam operation, as well.
At the bare minimum, my take is that he would be guilty of fraudulent bookkeeping, not having listed the bitcoin assets (as he probably knew a lot were missing). As to why they were missing, there are many theories to that – Karpeles stashing some, outside theft, or a combination of both.
But every single scenario ends in some kind of fraud.
Stored the first bitcoins I bought at m-t-Gox so I could work out a proper and safe cold-storage solution at home and they were portrayed as the veterans of bitcoin by many, but then suddenly discovered that I was no longer allowed to transfer them out, because of so called problems with my verification. Then followed months of stalling and refusing my verification until a few days after they stopped BTC-withdrawals all together(no new documents required btw). It now looks like a ploy to steal my btc and stall for time.
Any advise on how a EU-citizen would file a claim on mtGox? I’m not a wealthy person and I’m not talking about hundreds of coins, but that doesn’t mean I can just laugh that loss away. Any chance for small timers like me to still be able to lay a claim?
I’m also interested in filing a claim against mtGox from within EU.
I already signed up for this, but not sure how it will go:
http://www.mtgoxrecovery.com/
Perhaps also see http://www.selachii.co.uk/solicitors-blog/
But how did they lose all the fiat? Isn’t it strange that the fiat is gone as well? After all: It was kept in regular bank accounts?
And btw: Which exchanges, if any, do you trust now? All exchanges can claim, as Gox did, that their security is flawless. Until they’re mysteriously hacked.
What criteria do you now use to determine if an exchange is reliable with regards to their security claims?
Show me your source code, and I will show you my money.
Thanks, Rick.
Thanks for making my eyes bleed, that is.
Rick, in general you will find it behooves you to not comment on stuff you know nothing about.
I love your writing on stuff you do know a lot about, and wish you the best of luck moving past any losses.
You’re implying that there are factual errors in the article. May I ask where, so I can verify that assertion and correct them if appropriate?
Also, thanks for the kind words!
Cheers,
Rick
No need, I don’t think you are trying to write a compsec blog. What bothered me was implications that: folks at apple inc. would get sec right, Mark’s php ssh code went live on gox, roll-your-own is *always* bad, you know something about what happened to the gox coins.
Thanks for voicing your concerns, I’ll try to keep them in for future writing. As for your points:
> folks at apple inc. would get sec right
No, but they have a whole lotta more eyeballs
> Mark’s php ssh code went live on gox
Nobody knows this, but it’s not really relevant: the idea of making an SSH server in PHP is more of an illustration of his idea of good code. Also, we DO know that the Gox core code was built in PHP, which is more than bad enough. We know this because the source code has leaked.
> roll-your-own is *always* bad
As for security, in terms of replacing well-known implementations, I’d say it is. Other than that, I’ve been coding for… some 35 years, so I’ve certainly rolled my own substitutes for many things that haven’t met my requirements.
> you know something about what happened to the gox coins.
I wish. But no, no more than anyone else who follows /r/bitcoin.
Cheers,
Rick
Jesus christ, Rick. Are you for real you hadn’t done any background check on Empty Gox before putting your money in it? With that post about PHP/SSH, I’d expect it’d be widely known in the BTC community Gox was a time bomb. How come only now you’re posting something like that?
This is maddening fishy, mate…
Nope, I didn’t do any background checks. When I went in to bitcoin early 2011, Empty Gox was pretty much the only bitcoin exchange in operation, and it seemed to work well.
There hadn’t been any exchange options with any trade volumes worth mentioning before the summer of 2013, so it wasn’t a matter of doing research – if you wanted to play, and I did, then Gox was the exchange.
Seems a lot of us are paying the early-adopter fees right now.
Cheers,
Rick
This sounds like Jailhouse Rock.
You absolutely need to have your new Bud, Charlie Shrem check your facts before submission. I doubt he would appreciate your articles now. He took your speech and made it his – all he had to do was give you a high-five ONCE. Now you’re Buds.
He won’t be high-fiving this one, he doesn’t NEED you now. He took what he wanted and he’s long gone.
Stop looking at just Mt Gox but look to the entire Bitcoin Brotherhood or the Jeew Brotherhood (Charlie, Mark, etc.)
Adam ‘B’ Levine is trying to sell their bogus scam to the ‘Empty Gox community’ to take part in the new GOXCOINS. Saying we don’t need to know what happened at Mt Gox – we need to move on to this new coin and made whole at some point(?). It’s a scam for the ones who lost at MT Gox and MT GOX will wipe their slate clean – no more BTC debt. Awesome!
Adam says he gains nothing from the new Goxcoins BUT (a big BUT) someone is banking rolling him!
Your bud Charlie has been on overdrive recently condemning the Newsweek article but no worries about the inside scams.
It’s all scam but at least I’ve got lots of Kanelbullar to keep me happy.
the more i read about this the more i believe it’s a sting. it’s just like when mexico’s banpais went belly up. the owner lent himself a lot (A LOT) with no guarantees, “couldn’t” pay back, then fled the country. the government delayed everything for a couple years (some officials became richer in the process), then made it a national debt (FOBAPROA) and there was no one to blame. Some years later, the guy even sued the government for his losses and wanted his bank to be “saved” too.
i firmly believe this was a setup from karpeles, probably aided/pushed by some government(s), and no, i’m not paranoid, i just follow the ones who gain more, and it’s not the money (it can’t be easy to move THAT)
i agree that bitcoin, as much secure/anonymous/free internet access is a threat to the controlling powers of the world (which fit into the “aided by some government” part) and this is serious damage to the concept.
but again, i never though bitcoin would take. i always thought it was too new, but something would follow. bitcoin, like napster, led the charge but was the first to fall. hopefully, like napster, not in vain.
Why the fuss?
According to the REAL Satoshi:
” Instead of the supply changing to keep the value the same, the supply is predetermined and the value changes. As the number of users grows, the value per coin increases.”
http://web.archive.org/web/20090221024857/http://p2pfoundation.ning.com/forum/topics/bitcoin-open-source
Sounds like a pyramid scheme…. just wipe the slate clean and try again. It’s just a game anyhow. Right? Who cares if the security is lame?
We don’t know much about Gox’s security. You make a big show about the PHP SSH server, which probably did not run at MtGox (we have zero indication it did). But the fact that it was negligent and/or fraudulent was widely known for over a year.
In fact, the M-t-Gox (“empty gox”) joke is old. They had already been publicly hacked multiple times, and we don’t know how many times we never hard about it. Passwords were leaked, Bitcoins were lost. Most people know, and most people got their money out, as evidenced by the latest leak where only 70k of the 1M accounts had any Bitcoins left.
So I would be much more interested in why high profile Bitcoin enthusiasts such as Charlie Shrem vouched their full confidence for MtGox as late as this year. Why? What was their motivation? Are there any truths to the rumors that friends of Karpeles did indeed have ways to move money out and made a killing from arbitrage?
There’s also the fact that MtGox’s cold wallet has moved long after the so called “hack”. Only a few days ago someone ran their hot wallet preparation script. It couldn’t have been the hackers, because why would they wait over a month and risk Karpeles moving the stash away? A LOT of things are very very fishy about this.
Are you aware of any exchanges with transparent security practices that have been audited by security professionals?
I disagree with one point. Languages are neither secure nor insecure – programmers are. A programmer with extensive security experience can code a secure site in any language he/she is experienced in. One without security experience should not be coding a secure site in ANY language.
I don’t claim to be a security expert – but I have coded several secure sites in PHP. None of these sites has ever been hacked, despite attempts to do so. But that’s because I use good coding practices and check *EVERYTHING*.
This is obviously an inside job. To what magnitude others may be involved, I am unsure. Karpeles is a guilty little rat, and you know hes sitting on an anonymous stash of btc. Very strange movements on the blockchain recently. Huge amounts of coin. Pure and absolute negligence in so many separate ways on the part of Karpeles and Gox. Lack of info released reads just like a ponzi scheme. Larpeles has a criminal history in France. Wake up people!
One of the geniuses of bitcoin is that it is virtually impossible to distinguish between loss due to incompetence and loss due to theft. Turns out that hiding a few small ones and zeroes is far easier than hiding cash. Why wouldn’t Karpeles steal the BTC? There’s no way to tell.
If Karpeles has ~500K BTC , then every time you use bitcoin, you are enriching that asshole thief. If the 900K BTC was all stolen, then you are enriching some different thieves.
The only way to win is to ditch bitcon and move on to superior cryptocurrencies, ones that have superior security features that remove the need for centralization.
Well every time you use / save / invest ordinary money you enrich some bankers. Every time you use credit card you enrich some credit card company. There are plenty of people having called bankers thieves throughout history as well.
If you look back to 2009, bitcoin was actually started because of financial crisis which exposed some of the most serious flaws of the economic system up to then.
Heavy risk-taking. Because if they won, they could keep the money. If they lost, they knew the tax payers would be the ones paying for the gamble.
Some people will try to take advantage of the system. Bitcoin is not “perfect” – it’s a first try with crypto currencies. But it clearly does not have some of the bad properties which fiat money has.
I know that making own security things is bad practice and usually highly hackable, but how would new inventions in the field of security be made? That doctrine will mean that out of the thousand new ideas, not only the 999 more or less worthless ones will be discarded, but also the one super great best-security-ever one.
Using them in live action, as the only thing protecting something important, like Karpeles seem to have done, should of course never be done, but the rule doesn’t seem to be that. The rule seems to be ‘don’t even try, no matter what’. Seriously, whats wrong with coming up with homemade security and nesting it in regular security as an “extra skin”, or just publishing the idea cc0 for everyone to inspect, test and play around with?
New advancements in security get made by people who are experts at it and have thousands of other experts inspect it. Adding your own as an extra layer will frequently introduce more holes than it gets rid of. If you want to try, then yes, get experts to inspect it first, and don’t be surprised when the very, very quickly find problems or tell you that the idea has already been tried and found to be a bad one.
Noted, but now I’m just curious, how would adding extra that is flawed, compromise existing? I know we’re talking security generally, but ponder putting for example a simple Caesar cipher in a PGP e-mail. The Caesar cipher is of courze sillily easily crackable, but how would it compromise the secure PGP? Could something of todays strength, put in a PGP email, compromise it`?
Also question was not what to do if I want to try, which I couldn’t as I don’t know much at all about what’s involved, I’m not even a programmer in any modern languge, it was why there seems to be this doctrine like “never try, never say, never think” instead of just “never use for serious purpose unless thoroughly inspected and tested with good result”, like you (seem to) say.
You can poop a trillion times and still not make a space-worthy rocket ship.
Hey, ease down on the hate. Obviously someone or someones invented SHA, SSL, PGP an all what their abbreviations are. Most likely based on others research, based on others research, based on others research… And inspected by thousands, but still someone invented them. If everyone followed said doctrine we wouldn’t have those, or anything else.
I don’t claim to have an idea for anything new, but even if I did, why should I not do anything with it, especially just publish it? Such a comment, and such a doctrine, actually seems ideal for suppressing development.
As someone who is minimally involved in cryptocurrency at the moment, please enlighten me: how is it that–being the decentralized, peer-to-peer currency that cryptocurrency is–we need central exchanges?? Central point of failure and control, ahem?? Are things that bad? Is there no longer a reason to think Bitcoin is any different from the Us Dollar?
It’s not about the decentralized nature of the crypto-currency. It’s all about the human tendencies to believe.
It appears that mr Falkvinge fell victim to that. (I hope he has take some of his bitcoins out and didn’t lose all).
The best way to deal with bitcoins is to use the trading sites for trading while keeping most of your coins in a cold wallet (on a usb-stick or two at home). That way, most of your coins are safe, while enjoying the benefits of trading.
Most people will fall for the ‘trust us, we are safe’-lies that every company spouts. politicians not excempt.
[…] the nickname for the online accounts used to store bitcoins. Anybody who lost their money in the Mt. Gox debacle can attest to that. In many cases, all a scammer needs is to trick a user into revealing their “private key” to […]
A very good article. I am thinking of using this in my extended essay, but I want to make sure that falkvinge is a viable source. Any means of doing so?
Thank you. If you have a need to prove the references of an article written by me, I’d suggest linking to the articles about me on Wikipedia, deferring the judgment to the reader? Transparency always helps.
Given my credentials (named Top 100 Global Thinker, nominated by TIME as one of world’s 100 most influential, etc.), there should be something to hold on to.
Cheers,
Rick
What’s up colleagues, good piece of writing and nice arguments
commented here, I am in fact enjoying by these.